Setting apparatus, setting method, and storage medium

ABSTRACT

A setting apparatus comprises a setting unit that sets a virtual light parameter for applying an effect of irradiating virtual light on a plurality of subjects included in an image; and a classification unit that classifies the plurality of subjects into a plurality of groups based on a state of shadow. The setting unit sets the virtual light parameter for each group classified by the classification unit.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to a technique for correcting thebrightness of an input image.

Description of the Related. Art

Conventionally, a technique of relighting processing which irradiateslight from a virtual light source to a subject in a shot image is known.By performing the relighting processing, it is possible to brighten adark area such as shadow caused by an ambient light and obtain a desiredimage.

For example, Japanese Patent Laid-Open No. 2016-72694 disclosesrelighting processing that can appropriately correct a shadow of asubject. Specifically, the state of shadows of a predetermined area ofthe photographed image is detected, and characteristics of a virtuallight source are determined based on the detected state of shadows.Then, the shot image is corrected so that the state of shadows becomesthe state as when the subject is irradiated with light from the virtuallight source having the determined characteristics.

However, according to the method disclosed in the Japanese PatentLaid-Open No. 2016-72694, for example, in the case where a large numberof people are photographed as subjects, after detecting the state ofshadow of each subject, the characteristics of the virtual light sourcesuitable for each subject are determined. Accordingly, since thecharacteristics of the virtual light source are repeatedly calculatedfor the number of people, the time taken to perform relightingprocessing increases.

Further, even in an application in which the user can arbitrarily setthe parameters of the virtual light source, in the case where a largenumber of people are photographed as subjects, for example, it is verytroublesome for the user to set the virtual light source parameters foreach subject.

SUMMARY OF THE INVENTION

The present invention has been made in consideration of the abovesituation, and reduce the time taken to perform relighting processingeven in the case where a plurality of people are photographed assubjects.

According to the present invention, provided is a setting apparatuscomprising at least one processor and/or circuit configured to functionas following units: a setting unit that sets a virtual light parameterfor applying an effect of irradiating virtual light on a plurality ofsubjects included in an image; and a classification unit that classifiesthe plurality of subjects into a plurality of groups based on a state ofshadow, wherein the setting unit sets the virtual light parameter foreach group classified by the classification unit.

Further, according to the present invention, provided is a settingmethod comprising: classifying a plurality of subjects included in animage into a plurality of groups based on a state of shadow, setting avirtual light parameter for applying an effect of irradiating virtuallight on a plurality of subjects for each group classified by theclassification unit.

Furthermore, according to the present invention, provided is anon-transitory storage medium readable by a computer, the storage mediumstoring a program that is executable by the computer, wherein theprogram includes program code for causing the computer to function as asetting apparatus, comprising: a setting unit that sets a virtual lightparameter for applying an effect of irradiating virtual light on aplurality of subjects included in an image; and a classification unitthat classifies the plurality of subjects into a plurality of groupsbased on a state of shadow, wherein the setting unit sets the virtuallight parameter for each group classified by the classification unit.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention, andtogether with the description, serve to explain the principles of theinvention.

FIG. 1 is a block diagram showing a configuration of a digital cameraaccording to an embodiment of the present invention;

FIG. 2 is a block diagram showing a configuration of an image processingunit according to the embodiment;

FIG. 3 is a flowchart showing processing performed in the imageprocessing unit according to the embodiment;

FIG. 4 is a block diagram showing a configuration of a relightingprocessing unit according to the embodiment;

FIG. 5 is a schematic view for explaining reflection of virtual lightfrom a virtual light source according to the embodiment;

FIGS. 6A and 6B are explanatory diagrams of parameter setting of thevirtual light source according to the embodiment;

FIGS. 7A and 7B are views showing examples of images before and afterrelighting processing according to the embodiment;

FIG. 8 is a flowchart showing relighting processing according to theembodiment;

FIG. 9 is a flowchart showing the relighting processing according to theembodiment;

FIGS. 10A to 10D are diagrams illustrating the relighting processingperformed on a plurality of people according to the embodiment; and

FIG. 11 is a diagram for explaining grouping according to theembodiment.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present invention will be described indetail in accordance with the accompanying drawings. In this embodiment,an example in which the present invention is applied to a digital cameraas an image capturing apparatus will be described.

FIG. 1 is a block diagram illustrating an example of a configuration ofa digital camera 100 according to a first embodiment. In the digitalcamera 100 as shown in FIG. 1, light that has entered through a lensgroup 101 (an imaging optical system) that includes a zoom lens and afocus lens, and a shutter 102 having an aperture stop function,undergoes photoelectric conversion in an image sensing unit 103. Theimage sensing unit 103 is comprised of CCD, CMOS sensor, or the like,and an electric signal obtained by the photoelectric conversion isoutput to an A/D converter 104 as an image signal. The A/D converter 104converts an analog image signal output from the image sensing unit 103to a digital image signal (image data), and outputs the digital imagesignal to an image processing unit 105.

The image processing unit 105 applies to the image data from the A/Dconverter 104, or image data read out from an image memory 106 via amemory controller 107 a variety of image processes, such as a colorconversion process including white balance correction, γ correction, anedge enhancing process, color correction, and so on. The image dataoutput from the image processing unit 105 is written to the image memory106 via the memory controller 107. The image memory 106 stores imagedata output from the image processing unit 105 and image data to bedisplayed on a display unit 109.

A face/face parts detection unit 113 detects a face and a face-partregion where a human face and face parts exist in a captured image. Theimage processing unit 105 performs a predetermined evaluation valuecalculation process using the result of the face detection and theresult of the face-part detection by the face/face parts detection unit113 and the captured image data, and a system controller 50 performsexposure control and focus control using the obtained evaluation values.In this manner, through-the-lens auto focus (AF) processing, autoexposure (AE) processing, and auto white balance (AWB) processing, andso forth, are performed.

A D/A converter 108 converts the digital image data for display storedin the image memory 106 into an analog signal, and provides the analogsignal to the display unit 109. The display unit 109 displays an imageon a display screen, such as LCD, in accordance with the analog signalfrom the D/A converter 108.

A codec unit 110 compresses and encodes the image data stored in theimage memory 106 based on standards, such as JPEG and MPEG. The systemcontroller 50 stores the encoded image data to a recording medium 112,such as a memory card, hard disk, and so on, via an interface (I/F) 111.Further, image data read out from the recording medium 112 via the I/F111 is decoded and expanded by the codec unit 110, and stored in theimage memory 106. By displaying the image data stored in the imagememory 106 on the display unit 109 via the memory controller 107 and theD/A converter 108, an image can be reproduced and displayed.

A relighting processing unit 114 performs relighting processing thatcorrects brightness of a captured image by illuminating with a virtuallight source. The relighting processing performed by the relightingprocessing unit 114 will be explained later in detail.

The system controller 50 performs overall control of the digital camera100. A non-volatile memory 121 is configured with memory, such asEEPROM, and stores programs and parameters that are necessary, forprocessing by the system controller 50. Each process of the presentinvention as described later is realized by developing, in a systemmemory 122, programs, constants and variables stored in the non-volatilememory 121 for operation of the system controller 50 and executing theprograms.

An operation unit 120 receives operation, such as a menu setting, animage selection, and so on, by a user. A distance measuring sensor 123measures the distance to an object, and outputs distance informationcorresponding to each pixel of a captured image (distance informationdetection).

Next, details of the image processing unit 105 will be described withreference to FIGS. 2 and 3. FIG. 2 is a block diagram illustrating aconfiguration of the image processing unit 105, and FIG. 3 is aflowchart showing processing performed by the image processing unit 105.In this embodiment, it is assumed that the image sensing unit 103 iscovered with a color filter of Bayer arrangement. Accordingly, eachpixel of the image sensing unit 103 outputs one of R, G and B imagesignals.

First in step S301, image signals inputted from the A/D converter 104 inFIG. 1 is inputted to a synchronization processing unit 200. Thesynchronization processing unit 200 performs synchronization process oninputted R, G and B image signals to generate R, G and B color signalsfor each pixel. Next in step S302, a WB amplifying unit 201 appliesgains to the generated R, G and B color signals of each pixel based onwhite balance gains calculated by the system controller 50 in a knownprocess, thereby correcting white balance. The R, G and B color signalswhose white balance is corrected by the WB amplifying unit 201 areinputted to a luminance/chrominance signal generation unit 202 in stepS303. The luminance/chrominance signal generation unit 202 generates aluminance signal Y from the R, G and B color signals, and outputs thegenerated luminance signal Y to an edge enhancement unit 203, andoutputs the R, G and B color signals to a color conversion unit 205.

In step S304, the edge enhancement unit 203 performs an edge enhancementprocess on the luminance signal Y, and outputs the result to a luminancegamma processing unit 204. Meanwhile, the color conversion unit 205applies a matrix operation to the R, G and B color signals, convertsthem to have a desired color balance, and outputs the result to achrominance gamma processing unit 206 and a subject informationextraction unit 208.

In step S305, the subject information extraction unit 208 extractsinformation on the subject in a captured image based on the face/faceparts detection information output from the face/face parts detectionunit 113 and the R, G and B color signals output from the colorconversion unit 205 (subject information detection). Here, theinformation on the subject includes the number of subject/subjects inthe captured image, the size of each subject, the positionalrelationship between the subjects, how each subject is illuminated, theshadow information of each subject, and so on. For example, the number,size/sizes and positional relationship of the subject/subjects aredetected from the coordinate position information of each face/face partoutputted by the face/face parts detection unit 113, and how eachsubject is illuminated and the shadow information are detected from theentire captured image, the average luminance information and luminancehistogram information of each object.

In step S306, the luminance gamma processing unit 204 performs a gammaprocess on the luminance signal Y, and outputs the result to the imagememory 106 via the memory controller 107. Meanwhile, the chrominancegamma processing unit 206 performs gamma correction on the R, G and Bcolor signals, and outputs the result to a color difference signalgeneration unit 207. In step S307, the color difference signalgeneration unit 207 generates color difference signals R-Y and B-Y fromthe R, G and B signals, and outputs the result to the image memory 106via the memory controller 107.

Next, the relighting processing according to the present invention willbe explained. FIG. 4 is a block diagram illustrating the configurationof the relighting processing unit 114. The relighting processing unit114 reads the luminance signal Y and the color difference signals B-Yand R-Y processed by the image processing unit 105 and recorded in theimage memory 106, takes them as inputs, and perform relightingprocessing using a virtual light source.

First, an RGB signal conversion unit 401 converts the input luminancesignal Y and the color difference signals R-Y and B-Y into R, G and Bsignals, and outputs the result to a de-gamma processing unit 402. Thede-gamma processing unit 402 performs an operation (de-gamma processing)whose characteristics are opposite to those of the gamma correctionperformed by the luminance gamma processing unit 204 and the chrominancegamma processing unit 206 of the image processing unit 105, and convertsthe R, G and B signals to linear data. The de-gamma processing unit 402outputs the R, G and B signals that are converted to the linear data(Rt, Gt and Bt) to a virtual light source reflected componentcalculation unit 406 and a virtual light source applying processing unit407.

On the other hand, a distance calculation unit 403 calculates a distancemap based on the distance information of the subject acquired from thedistance measuring sensor 123. The distance information of the subjectis two-dimensional distance information obtained for each pixel of thecaptured image. A normal vector calculation unit 404 calculates a normalvector map as shape information representing the shape of the subjectfrom the distance map calculated by the distance calculation unit 403.As a method of generating a normal vector map from the distance map, aknown technique is used, but a specific processing example will bedescribed with reference to FIG. 5.

FIG. 5 is a diagram showing the relationship between the image shootingcoordinates with respect to the camera 100 and a subject. For example,with respect to the subject 501 as shown in FIG. 5, from a differenceΔDH of the distance D with respect to a difference ΔH of a capturedimage in the horizontal direction and, although not shown, a differenceΔDV of the distance D with respect to a difference ΔV of the capturedimage in the vertical direction, inclination information of a part ofthe subject 501 is calculated. Then, a normal vector N is calculatedfrom the calculated inclination information of the part of the subject.By performing the above processing on each pixel of the captured image,it is possible to calculate the normal vector N corresponding to eachpixel of the captured image. The normal vector calculation unit 404outputs the normal vector N corresponding to each pixel of the capturedimage to the virtual light source reflected component calculation unit406 as the normal vector map.

Although the distance calculation unit 403 and the normal vectorcalculation unit 404 are described as being configured in the relightingprocessing unit 114, the present invention is not limited to this, andfor example, may be configured in the distance measuring sensor 123 orthe image processing unit 105, or may be configured independently.

A virtual light source setting unit 405 sets the parameters of a virtuallight source based on the subject information inputted from the subjectinformation extraction unit 208 of the image processing unit 105. Forexample, in a case where it is desired to increase the brightness of theentire face of a subject whose entire face is dark, parameters such asthe position, the irradiation range, and the intensity of the virtuallight source are set such that the entire face is included in theirradiation range of the virtual light source.

Here, with reference to FIG. 6A and FIG. 6B, parameters to be set for avirtual light source will be described taking a case where the subjectis one person as an example. FIG. 6A is a perspective view showing thepositional relationship between the subject and the virtual lightsource, and FIG. 6B is a plan view showing the positional relationshipbetween the subject and the virtual light source. Regarding the positionof the virtual light source, if the distance between the virtual lightsource and the subject is set to be short, the light of the virtuallight source strongly hits the subject, and conversely if the distanceto the subject is set to be long, the light of the virtual light sourcehit the subject weakly. Regarding the irradiation range of the virtuallight source, if the irradiation range of the virtual light source isset to be wide, light can illuminate the entire subject. Conversely, ifthe irradiation range is set to be narrow, light can illuminate only apart of the subject. Regarding the intensity of the virtual lightsource, if the intensity of the virtual light source is set to bestrong, light strongly illuminates the subject, and conversely if theintensity is set to be weak, light weakly illuminates the subject.

In the virtual light source reflected component calculation unit 406,based on the distance K between the light source and the subject, thenormal information N and the parameters of the virtual light source setby and the virtual light source setting unit 405, light virtuallyirradiated from the set virtual light source, the component which willbe reflected by the subject is calculated. Hereinafter, light virtuallyirradiated from the virtual light source is called “virtual light”.Specifically, the reflected components of the virtual light at the partof the subject corresponding to the coordinate position of the capturedimage is calculated so that the reflected components are inverselyproportional to the square of the distance K between the virtual lightsource and the part of the subject corresponding to each pixel and isproportional to the inner product of the vector of the normal N and thevector of the light source direction L.

Here, a general calculation method of the reflected components ofvirtual light will be described with reference to FIG. 5. In FIG. 5, forthe sake of simplicity of explanation, only the horizontal direction ofthe captured image is shown, but as described above, the directionperpendicular to the drawing is the vertical direction of the capturedimage. In the following description, a method of calculating thereflected components of virtual light at the point P1 on the subject 501corresponding to the horizontal pixel position H1 and the vertical pixelposition V1 (not shown) in the captured image will be described. In FIG.5, the virtual light source 502 is set for the subject 501. Thereflected components of the virtual light at the position (H1, V1) ofthe captured image shot by the camera 100 is proportional to the innerproduct of the normal vector N1 at the point P1 on the subject 501 andthe light source direction vector L1 of the virtual light source 502,and inversely proportional to the square of the distance K1 between thevirtual light source 502 and the point P1. The normal vector N1 and thelight source direction vector L1 are three-dimensional vectorsconsisting of a horizontal direction, a vertical direction, and a depthdirection (the direction indicated by the distance D in FIG. 5). Whenthis relationship is expressed by a mathematical expression, thereflected components (Ra, Ga, Ba) of virtual light at the point P1 onthe subject 501 are as shown by the following expressions (1).

Ra=α×(−L1·N1)/K1² ×Rt

Ga=α×(−L1·N1)/K1² ×Gt

Ba=α×(−L1·N1)/K1² ×Bt  (1)

Here, α is the intensity of light from the virtual light source, thegain value of the rewriting correction amount, Rt, Gt, Bt are the RGBsignals output from the de-gamma processing unit 402.

The reflected components (Ra, Ga, Ba) of the virtual light calculated asdescribed above are output to the virtual light source applyingprocessing unit 407. In the virtual light source addition applying unit407, the processing shown by the following expressions is performed inwhich reflected components (Ra, Ga, Ba) of the virtual light are addedto the R, G and B signals output from the de-gamma processing unit 402.

Rout=Rt+Ra

Gout=Gt+Ga

Bout=Bt+Ba  (2)

The R, B and G signals (Rout, Gout, Bout) which have undergone therelighting processed by the virtual light source applying processingunit 407 are input to a gamma processing unit 408 where gamma correctionis performed. Then, a luminance/color difference signal generation unit409 generates and outputs the luminance signal Y and the colordifference signals R-Y and B-Y signals from the gamma-processed R, G andB signals (R′out, G′out, B′out).

An example of the relighting processing described above in therelighting processing unit 114 is shown in FIGS. 7A and 7B. FIG. 7A isan example of a captured image before relighting processing, and FIG. 7Bis an example of the captured image after relighting processing. Thesubject which is dark as shown in FIG. 7A is corrected to be bright asshown in FIG. 7B by performing relighting processing which appliesvirtual light.

The system controller 50 accumulates the luminance signal Y and thecolor difference signals R-Y and B-Y output from the relightingprocessing unit 114 in the image memory 106 under the control of thememory controller 107 and then the codec unit 110 compresses and encodesthem. In addition, The processed signals are recorded in the recordingmedium 112 via the I/F 111.

Next, the relighting processing by the relighting processing unit 114according to the present embodiment will be described with reference tothe flowcharts of FIGS. 8 and 9. This processing is performed on theimage processed by the image processing unit 105 and stored in the imagememory 106 (namely, on the luminance signal Y and the color differencesignals R-Y and B-Y) when the relighting processing is selected by anoperation from a user via the operation unit 120.

First, in step S801, the virtual light source setting unit 405 acquiresthe subject information, acquired by the subject information extractionunit 208 in step S305, such as the number, size, and positionalrelationship of the subject/subjects included in the image subjected tothe relighting processing. In step S802, it is determined whether thenumber of subject/subjects is one or more. In the case of one person,the process proceeds to step S803, and in the case of plural people, theprocess proceeds to step S903 in FIG. 9.

In step S803, the distance calculation unit 403 generates a weightingmap (mapK) depending on the distance between the digital camera 100 andthe subject with respect to the area of the detected subject.Specifically, first, the distance calculation unit 403 calculates thedistance K on a pixel-by-pixel basis (distance map) based on thetwo-dimensional subject distance information obtained pixel by pixel ofthe captured image acquired from the distance measuring sensor 123.Then, a value obtained by normalizing 1/(K²) with an arbitrary bit widthon a pixel-by-pixel basis is defined as a distance weighting map (mapK).

Next, in step S804, a normal vector map (mapN) is generated with respectto the area of the detected subject in the normal vector calculationunit 404 based on the distance map acquired from the distancecalculation unit 403. Specifically, as described with reference to FIG.5, the subject normal vector N is calculated on a pixel-by-pixel basisand its direction cosine with respect to each coordinate axis directionis calculated. Then, the obtained direction cosine for each pixel isrepresented by an arbitrary bit width, and it is taken as the normalweighting map (mapN).

In step S805, the virtual light source setting unit 405 generates avirtual light weighting map for the detected subject region. Morespecifically, the light source direction vector −L of the virtual lightsource is calculated for each pixel, and the direction cosine of thevector −L for each coordinate axis direction is calculated. Then, theobtained direction cosine is represented by an arbitrary bit width on apixel-by-pixel basis, and taken as a weighting map (mapL) depending onthe virtual light source. The parameter setting of the virtual lightsource for calculating the light source direction vector −L of thevirtual light source is determined based on the subject informationinputted from the subject information extraction unit 208. For example,when the luminance distribution in the acquired face region is biased,the position, the irradiation range, and the intensity of the virtuallight source are determined so that the virtual light hits the regionhaving the low luminance value.

For example, if the coordinates in the captured image of the region withlow luminance value are (x1, y1), the reflected components (Ra (x1, y1),Ga (x1, y1), Ba (x1, y1), Ba)) are expressed by the followingexpressions (3).

Ra(x1,y1)=α×(−L(x1,y1)·N(x1,y1))/K(x1,y1)² ×Rt

Ga(x1,y1)=α×(−L(x1,y1)·N(x1,y1))/K(x1,y1)² ×Gt

Ba(x1,y1)=α×(−L(x1,y1)·N(x1,y1))/K(x1,y1)² ×Bt  (3)

In the equation (3), α is the light intensity of the virtual lightsource. L(x1, y1) is the light source direction vector of the virtuallight source at the position on the subject corresponding to thecoordinate (x1, y1), N(x1, y1) is the normal vector at the position onthe subject corresponding to the coordinate (x1, y1). Further, K(x1, y1)indicates the distance between the virtual light source and the positionon the subject corresponding to the coordinates (x1, y1). In order thatthe virtual light illuminate the subject at the coordinates (x1, y1)located in an area where the luminance value is row, the intensity α ofthe virtual light source and the distance K(x1, y1) to the subject arecontrolled so that (Ra(x1, y1), Ga(x1, y1), Ba(x1, y1)) have positivevalues. Also, coordinate information having a luminance value lower thanan arbitrary threshold value is acquired based on the acquired luminancedistribution information in the face area, and the irradiation range ofthe virtual light source is controlled so as to include the coordinatesin the coordinate information. Further, since there is a possibilitythat harmful effects such as high-light detail loss and tone valueinversion may occur if the intensity α of the virtual light source isset too high, the range of the intensity α of the virtual light sourceis controlled within a given range β with respect to ±β of an averageluminance value in a high luminance region outside the irradiationrange. Through the processing described above, the virtual light sourcesetting unit 405 calculates the range of the position of the virtuallight source, the range to be irradiated, the range of the intensity,and determines the setting values by taking the average values withinthe respective ranges as the position of the virtual light source, theirradiation range, and the intensity.

In step S806, the reflected components (Ra, Ga, Ba) of the virtual lightis calculated for the area of the detected subject in the virtual lightsource reflected component calculation unit 406. The reflectedcomponents (Ra, Ga, Ba) can be calculated using the expressions (1) asdescribed above. The expressions (1) are replaced by the weighting map(mapK) depending on the distance obtained in step S803, the weightingmap (mapN) depending on the normal of the subject obtained in step S804,and the weighting map (mapL) depending on the virtual light sourceobtained in step S805. In other words, the reflected components ofvirtual light can be calculated using the following expressions (4).

Ra=α×mapL·mapN·mapK×Rt

Ga=α×mapL·mapN·mapK×Gt

Ba=α×mapL·mapN·mapK×Bt  (4)

in step S807, the relighting processing is performed. Specifically, asshown in the expressions (2) above, the reflected components (Ra, Ga,Ba) of the virtual light calculated in step S806 are added to theoutputs (Rt, Gt, Bt) from the de-gamma processing unit 402 in thevirtual light source applying processing unit 407. Upon completion ofthe relighting processing, the processing by the relighting processingunit 114 is terminated.

As a result of the above processing, the relighting processing unit 114performs relighting processing when there is only one subject in thecaptured image.

On the other hand, if it is determined in step S802 that the number ofpeople of the subject is plural, the process proceeds to step S903 inFIG. 9. In step S903, the distance calculation unit 403 generates aweighting map (mapK) depending on the distance between the digitalcamera 100 and the subject in the same manner as in step S803. At thistime, a weighting map (mapK) depending on distance is generated for oneof the plurality of detected subjects.

Next, in step S904, similarly to step S804, a normal vector calculationunit 404 generates a normal weighting map (mapN). At this time, thenormal weighting map (mapN) is generated for the same subject as in stepS903.

In step S905, it is determined whether or not the processes in stepsS903 and S904 are completed by the number of detected subjects. If notcompleted, the process returns to step S903, processes are performed foranother person, and if completed, the process proceeds to step S906.

In step S906, the virtual light source setting unit 405 divides thesubjects in the captured image into groups. Specifically, the subjectsare divided into groups based on the subject information (the number ofsubjects in the captured image, the sizes of the subjects, thepositional relationship between the subjects, the shadow information ofthe subjects) acquired in step S801.

Here, the grouping of the subjects performed in step S906 will bedescribed with reference to FIGS. 10A to 10D and FIG. 11. FIGS. 10A to10D show examples of a captured image in the case where the capturedimage includes a plurality of subjects. FIG. 10A is a captured imagebefore relighting processing, and FIG. 10B shows a conceptual diagram inwhich the subjects are grouped in the virtual light source setting unit405. As shown in FIG. 10B, the grouping of the subjects is adjusted suchthat the subjects that are close to each other and have the same shadowinformation belong to the same subject group.

In step S907, it is determined whether or not the number of subjectgroups set in the virtual light source setting unit 405 in step S906 isequal to or less than an upper limit value. If it is equal to or lessthan the upper limit value, the process advances to step S908, and if itexceeds the upper limit value, the process returns to step S906 andadjusts the number of groups so as to be equal to or less than the upperlimit value.

Note that after dividing the subjects into the subject group, whichsubject has been classified into which group may be displayed on thedisplay unit 109 so that the user can identify the grouping result. As adisplay example at this time, for example, as shown in FIG. 10B, eachgroup may be surrounded by a frame and a group name may be displayed.Alternatively, indicators (for example, group name, icon, color frame)that indicate groups may be displayed for each subject.

Further, the user may change the result automatically classified by thedigital camera 100. In that case, the user inputs an instruction tochange the classification result via the operation unit 120. Variousmethods are conceivable as the changing method. For example, when thegroup is surrounded by a frame as shown in FIG. 10B, for example, bychanging the size of the frame, the subject included in each frame canbe changed. Alternatively, at least one subject may be selected by auser operation, and the group to which the selected user is classifiedmay be specified.

In step S908, the virtual light source setting unit 405 sets parameterssuch as the position, irradiation range, and intensity of the virtuallight source for each subject group determined in step S906. Parameterssuch as the position, irradiation range, and intensity of the virtuallight source are set in the same manner as in the above-described stepS805, where there is a difference in that the parameters are set for onesubject in step S805, but the parameters are set for each subject groupin step S908. Then, based on the parameters of the virtual light sourceset for each subject group, a weighting map depending on the virtuallight source is generated. More specifically, the light source directionvector −L of the virtual light source is calculated for each pixel, andthe direction cosine for each coordinate axis direction is calculated.The obtained direction cosine is represented by an arbitrary bit widthand taken as the weighting map (map L) depending on the virtual lightsource.

FIG. 11 shows an example of a table of virtual light source parametersettings used in the virtual light source setting unit 405 in step S908.Reference numeral 1101 denotes the subject No.; 1102, coordinates of thecenter of a face of the subject; 1103, the shadow information; and 1104,the subject grouping information. Incidentally, in FIG. 11, the subjectsin the front row of FIGS. 10A to 10D are labeled by 1 to 5 in order fromthe left side, and the subjects in the rear row are labeled by 6 to 12in order from the left side. Subject No. and the coordinates of thecenter of the face are acquired from the detection result of theface/face parts detection unit 113, and the shadow information isacquired from the luminance distribution information in the face area.The grouping of the subjects is adjusted such that the subjects that areclose to each other and have the same shadow information belong to thesame subject group. In the example of FIG. 11, subjects are classifiedinto three groups; group I for a subject whose whole face is bright,group II for a subject whose right upper part of the face is dark, andgroup III for a subject whose entire face is dark.

In step S909, using the parameters of the virtual light source set foreach subject group determined in step S906, reflected components ofvirtual light are calculated in the virtual light source reflectedcomponent calculation unit 406. The reflected components (Ra, Ga, Ba) ofthe virtual light can be calculated by the above-described expressions(3). For each subject group, the expressions (3) are replaced by theweighting map (mapK) depending on the distance generated in step S903,the weighting map (mapN) depending on the normal of the subjectgenerated in step S904, and the weighting map (mapL) depending on thevirtual light source generated in step S908. Thus, the reflectedcomponents of the virtual light are calculated using the above-describedexpressions (4).

FIG. 10C is a conceptual diagram of the calculation result of thereflected components of the virtual light calculated by the virtuallight source reflected component calculation unit 406. The calculationresult of the virtual light reflected components shows themultiplication result of the gain a (intensity), the weighting map (mapL) depending on the virtual light source, the normal weighting map (mapN), and the weighting map (mapK) depending on the distance as describedabove.

In step S910, the reflected components of the virtual light calculatedfor each object group in step S909 are synthesized in the virtual lightsource reflected component calculation unit 406 to calculate reflectedcomponents (Ra, Ga, Ba) of virtual light from the virtual light sourceof the entire image. For example, in the example of the captured imageshown in FIGS. 10A to 10D, the reflected components of virtual light ofthe subject groups I, II, and III are synthesized. A conventionalsynthesis method may be used here, as an example, a lighten compositeprocess or the like may be used.

In step S911, the relighting processing is performed. Here, therelighting processing is performed using the formula (2) as in stepS807, by adding the reflected components (Ra, Ga, Ba) of the virtuallight calculated by the virtual light source reflected componentcalculation unit 406 in step S910 with the outputs (Rt, Gt, Bt) of thede-gamma processing unit 402 in the virtual light source applyingprocessing unit 407.

FIG. 10D is an example of the captured image after the relightingprocessing. In FIG. 10A, the faces of the subjects in the shadow aredark, whereas in FIG. 10D, all subjects are corrected brightly by therelighting processing.

In the case where the image after the relighting processing is displayedon the display unit 109, it may be displayed so that a user can identifywhich object is classified into which group when the relightingprocessing is performed. Furthermore, the user may change the displayedclassification result, and the relighting processing may be redone. Inthis way, if the desired result of the relighting processing is notobtained, the user can redo the relighting processing.

With the above processing, the relighting processing unit 114 canperform the relighting processing on multiple subjects without taking along time.

In the processes shown in FIGS. 8 and 9, it is described that theweighting map depending on the virtual light source is generated aftergeneration of the weighting map depending on the distance and the normalweighting map, however, the order of generation is not limited thereto.After generating the weighting map depending on the virtual lightsource, the weighting map depending on distance and the normal weightingmap may be generated, or these weighting maps may be generated inparallel.

Further, in the present embodiment, the digital camera 100 has beendescribed as an example of the image capturing apparatus, however, thepresent invention can also be applied to an image processing device suchas a personal computer. In that case, the image processing device mayacquire an image captured by imaging means such as a camera, the usermay arbitrarily set virtual light source parameters with respect to theacquired image, and the image processing apparatus may perform therelighting processing. If there is additional information such as facedetection result, distance information, normal information, and so on,together with the image, the relighting processing may be performedusing such information. Even in that case, grouping the subjects makesit unnecessary for the user to set the parameters of the virtual lightsource for each subject, so it is possible to perform the relightingprocessing without complicating the parameter setting method of thevirtual light source.

Further, in the present embodiment, the case where there is one virtuallight source for each subject or group has been described, however, thepresent invention is not limited to this. For example, it is configuredto use a plurality of virtual light sources to perform the relightingprocessing such that one virtual light source may illuminate a subjectfrom the upper right position of the subject and another virtual lightsource may illuminate the subject from the side of the subject.

Further, in the present embodiment, the case of correcting a subject tobecome brighter using an added light has been described, however,relighting for making a subject darker may be performed. In that case,the gain value a of the virtual light source is set to a minus value(subtracted light). In addition, specular reflection light may be addedto the subject. In this way, any one of a plurality of types of lightcan be selected as virtual light.

Further, the method of calculating the distance D between the positionof the virtual light source and the processing target pixel is notlimited to the present embodiment, and any calculation method may beadopted. For example, the position of the camera and the position of thesubject may be acquired as a three-dimensional position and a distancemay be calculated in three dimensions.

In addition, in the case of adding virtual light, its intensity iscalculated using an equation inversely proportional to the square of thedistance, however, the present invention is not limited to calculatingthe addition amount of the virtual light by this method. For example, anequation which is inversely proportional to the distance D or anexpression in which the irradiation range varies in a Gaussiandistribution may be used.

In the above example, the case of detecting a person as a subject hasbeen described, however, the subject is not limited to a person, and apredetermined subject (for example, a car, an animal, a plant, etc.) maybe detected.

OTHER EMBODIMENTS

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Applications No.2018-029049, filed on Feb. 21, 2018, and 2019-018259, filed on Feb. 4,2019, which are hereby incorporated by reference herein in theirentirety.

What is claimed is:
 1. A setting apparatus comprising at least oneprocessor and/or circuit configured to function as following units: asetting unit that sets a virtual light parameter for applying an effectof irradiating virtual light on a plurality of subjects included in animage; and a classification unit that classifies the plurality ofsubjects into a plurality of groups based on a state of shadow, whereinthe setting unit sets the virtual light parameter for each groupclassified by the classification unit.
 2. The setting apparatusaccording to claim 1, wherein the at least one processor and/or circuitis configured to function as an image processing unit that applies aneffect of irradiating virtual light on the plurality of subjects basedon the virtual light parameter set by the setting unit.
 3. The settingapparatus according to claim 2, wherein the at least one processorand/or circuit is configured to function as a shape acquisition unitthat acquires shape information of the plurality of subjects, whereinthe image processing unit applies an effect of irradiating virtual lighton the plurality of subjects based on the virtual light parameter andthe shape information.
 4. The setting apparatus according to claim 3,wherein the at least one processor and/or circuit is configured tofunction as a distance acquisition unit that acquires distanceinformation of the plurality of subjects, wherein the image processingunit applies an effect of irradiating virtual light on the plurality ofsubjects based on the virtual light parameter, the shape information,and the distance information.
 5. The setting apparatus according toclaim 3, wherein the at least one processor and/or circuit is configuredto function as a distance acquisition unit that acquires distanceinformation of the plurality of subjects, wherein the shape acquisitionunit acquires shape information based on the distance information. 6.The setting apparatus according to claim 1, wherein the virtual lightparameter includes an irradiation range and intensity of the virtuallight.
 7. The setting apparatus according to claim 1, wherein thesetting unit sets the virtual light parameter for each group based on auser operation.
 8. The setting apparatus according to claim 1, whereinthe virtual light includes addition light that adds brightness to asubject, subtraction light that darkens a subject, specular reflectionlight that applies specular reflection effect on a subject.
 9. Thesetting apparatus according to claim 1, wherein the at least oneprocessor and/or circuit is configured to function as an image sensorthat shoots a subject and obtains image data.
 10. The setting apparatusaccording to claim 1, wherein the at least one processor and/or circuitis configured to function as a display controller that controls todisplay a result of the classification by the classification unit on adisplay.
 11. The setting apparatus according to claim 1, wherein the atleast one processor and/or circuit is configured to function as an inputunit that accepts an instruction to change a result of theclassification by the classification unit.
 12. The setting apparatusaccording to claim 11, wherein the input unit accepts an instruction forselecting at least one subject and classifying the selected least onesubject into a designated group.
 13. The setting apparatus according toclaim 2, wherein the at least one processor and/or circuit is configuredto function as a display controller that controls to display a result ofthe classification by the classification unit on a display, wherein theresult of the classification by the classification unit is displayedbefore applying the effect of irradiating the virtual light by the imageprocessing unit.
 14. The setting apparatus according to claim 2, whereinthe at least one processor and/or circuit is configured to function as adisplay controller that controls to display a result of theclassification by the classification unit on a display, wherein theresult of the classification by the classification unit is displayedafter applying the effect of irradiating the virtual light by the imageprocessing unit.
 15. The setting apparatus according to claim 14,wherein the at least one processor and/or circuit is configured tofunction as an input unit that accepts an instruction to change theresult of the classification by the classification unit.
 16. A settingmethod comprising: classifying a plurality of subjects included in animage into a plurality of groups based on a state of shadow, setting avirtual light parameter for applying an effect of irradiating virtuallight on a plurality of subjects for each group classified by theclassification unit.
 17. A non-transitory storage medium readable by acomputer, the storage medium storing a program that is executable by thecomputer, wherein the program includes program code for causing thecomputer to function as a setting apparatus, comprising: a setting unitthat sets a virtual light parameter for applying an effect ofirradiating virtual light on a plurality of subjects included in animage; and a classification unit that classifies the plurality ofsubjects into a plurality of groups based on a state of shadow, whereinthe setting unit sets the virtual light parameter for each groupclassified by the classification unit.